Distinguishing Humans from Robots in Web Search Logs

The workload on web search engines is actually multiclass, being derived from the activities of both human users and automated robots. It is important to distinguish between these two classes in order to reliably characterize human web search behavior, and to study the effect of robot activity. We suggest an approach based on a multidimensional characterization of search sessions, and take first steps towards implementing it. We present a few behavioral criterions, such as the maximal number of queries in a day, query submittal rate, minimal interval of time between different queries, and more. By studying the interaction between the behavioral criterions of the humans vs. the robots, we were able to classify the users in the log files. In order to find the best classification, we used a grading method that helped us to compare the results from several analysis. Our conclusion was that analysis that combines a few criterions may give the best classification between humans and robots.